AI reshapes economies and daily life; this session navigates ethical tensions and asks what values should anchor decision‑making in an AI‑driven age.
In a Davos town hall on “Dilemmas around Ethics in AI,” panelists argued that today’s core AI problems are less about abstract “trust” and more about power, safety, and governance. Rachel Botsman challenged the premise of the opening poll: “Trust is useless without context,” warning that pervasive automation can produce “cognitive atrophy” as people lose tolerance for uncertainty and outsource judgment. Max Tegmark framed the ethical crisis as regulatory exceptionalism: AI is “less regulated even than sandwich shops,” despite harms such as chatbot-driven manipulation and child suicides. He urged “binding safety standards,” akin to the FDA or crash testing, to force companies to make the safety case before deployment. Meredith Whittaker rejected “ethics” as “paint brushes… on the side of the bomber,” arguing the industry’s deep-learning “monoculture” is a contingency of platform monopolies, data concentration, and massive capex incentives. She also highlighted environmental and creative extraction, and warned that OS-level “AI agents” seeking broad permissions resemble “the architectures of targeted malware,” undermining secure communications and user agency. Across views, the panel converged on a pragmatic agenda: treat AI like other high-risk industries, demand transparency, and resist inevitability narratives that substitute acceleration for accountability.
Hello, everybody. Welcome to the World Economic Forum's Town hall on dilemmas around ethics and AI. My name is Matt Honan. I'm the editor in chief of MIT Technology Review. We're joined today by Meredith Whittaker, who's the president of the Signal Foundation and the co-founder of the AI Now Institute, Max Tegmark, an MIT, Max Tegmark, an MIT professor with who teaches on AI and also is the co-founder of the Future of Life Institute. And Rachel Botsman, an author, author, author, author, artist, and expert on trust in the digital age and an associate fellow at Oxford University. Welcome, everybody. Thanks for joining us. This is designed to be a town hall setting. We want you to be involved in all of this. We'd like for you to ask questions. We're going to have polls that you can answer. We really want this to be an interactive session, and we hope that you will join us. And so to start with, you'll see up here a QR code that you can all look at and scan with your phones. It will launch something called Slido. And we've got a poll to kick off for all of you that asks, how much do you trust AI systems in your daily lives? And so that's for you to answer, but I'd like for each of you here on our panel to answer something for me, which is maybe you can tell me what you see as the most urgent ethical challenge facing us today when it comes to AI. Rachel, would you like to kick us off?
Yeah. I actually think that poll question is a terrible question.
Oh, no.
Sorry. It's. I think it's part of the challenge. And I've listened to it all week because, trust is it's useless without context. So that question, do you trust AI in your life? I would say to do what? And you can't really have an ethical debate until you think about context. And this is missing from so many conversations. You know, there's certain things I definitely trust AI to do. And then there's things I don't. And until you think about context, it's really hard to frame the issue. So please do answer the polling question. Because I'd be interested to hear the results. But I thought what I could bring is I could talk about trust and systems and trust in platforms and talk about this at a societal level. But the thing I'm actually really passionate about is, I write and I make art, and so I'm very connected to the hand mind connection. And one of the things that worries me is as human beings, we're very bad at recognizing when we're outsourcing thinking, and when that thinking becomes so efficient that we literally stop thinking. And so cognitive atrophy when that sets in. I already see it in peers and in students. Not just the speed at which they work, the speed at which they expect answers, but it's also their tolerance for friction, their ability to sit with doubt and uncertainty. And this worries me for someone whose job it is to think, but also because to make. So. I know that when you make art, AI can do incredible things. I use it in my work to model things to get things out my head. But that hand mind connection is so important. So when that goes and we outsource that to something that is deciding what we care about, what we're interested in, what we should be curious about, which at the end of the day, it doesn't care about us, it does not care about us, but it's shaping those decisions. I really worry about that. I really worry about that in children, how we don't lose that connection and how as human beings, we do a much better job at parsing. When is it great for outsourcing thinking? Because it frees up our cognitive abilities and creates that space for many wonderful things and solves problems? And when does it create cognitive atrophy? So that's that's my concern, Max.
If, we set different rules for people of different skin color, that would be viewed as very, very unethical. And I feel we're doing exactly that with companies from different industries. Now, if you are a drug company, for example, and you want to sell a new kind of pill, which might increase suicidal ideation in teenagers, you have rules. You're not even allowed to sell your first pill until you've done a clinical trial and measured the suicidal ideation. And and if it's problematic, you go back to the drawing board. Yet it's completely legal in America now, for a company to sell an AI girlfriend to 12 year olds, even though we know that that has increased suicidal ideation and caused many, many young kids to take their own lives. So massive discrimination against where somehow we've decided that AI as an industry in America is actually less regulated even than sandwich shops. You know, if if you want to open a cafe before you can sell your first sandwich, you have to have the health inspector come in and check. And if he finds 53 rats in your kitchen, you're not selling any sandwiches today. And if. But if you then turn around and say, you know, don't worry, I'm not going to sell any sandwiches. I'm just going to sell chatbots, AI, girlfriends for 12 year olds, or I'm just going to build superintelligence, which some of the AI companies are stating openly is their goal. While they're saying Elon Musk was on a stage last month saying, oh, you know, it's going to be the machines in charge, not the people. And other CEOs have talked about. We're going to build a new robot species, which is going to replace us. Oh no problem, as long as you don't sell sandwiches. So I think we urgently need to cut this corporate welfare, you know, and start treating the AI industry in the same way we treat any other industry with safe, binding safety standards.
Meredith.
Yeah. You know, I think that sort of the term ethical here is a, you know, it's kind of a dog whistle that means these things are going to be treated as a sort of afterthought. And, you know, the fundamental work of building these systems, that's the serious work. And then let's come with some paint brushes and paint some smiley faces on the side of the bomber. And I think that that framing kind of tells us where we are in some sense, because what we're talking about or what I'm talking about, what I think we're looking at are fundamental limitations, fundamental risks, and, you know, real issues of like, who and how and where are these systems being created and who gets to determine on whom they are used, how they are used, what they do. And this is not these aren't afterthoughts, right? They're fundamental foundations on which the house of AI is being built.
Can you.
Talk to me a little about what those fundamental risks are like? What are they entail for us?
I mean, I'm not going to stack rank them. But, you know, I think I think very quickly, if we look at what the AI industry is right now, it's a it's a monoculture. It is focused. It's almost exclusively, at least in the Davos level, popular discourse on deep learning based, language centric llms. That's one approach to AI that over its 80 year history has, you know, has shown up a couple times. But there are many, many other approaches to using data and using computational systems to create AI. You know, AI approaches, right? The reason deep learning based, you know, these sort of LLM centric models are now everywhere is due to a business model that emerged out of the commercialization of computation of the internet in the 1990s, enabled large platforms to build sort of monopolistic communications and platform networks. That and, you know, communication networks are natural monopolies. We know this from the telephone. We know this from the telegraph. You know, these these forms are no different. Build these large sort of data based, you know, kind of advertising based platforms that gathered a huge amount of data and that built out infrastructures to process and and use that data. So indexing ads, indexing, search, you know, the Facebook, social media model. So why am I even talking about that? That's old news. We're in the AI era, right? I'm talking about that because that those are the conditions in which deep learning became newly interesting in the early 20 tens, before we were all still calling it machine learning. We there was a recognition that old techniques from the 1980s and the early 90s could do new things that were newly interesting, particularly to indexing social media feeds and sort of optimizing for engagement when they were matched with those resources, data and compute. What is new in this era are not the deep learning centric approaches. It is the access to huge amounts of data and huge amounts of compute that are pooled in the hands of a handful of actors due to their establishing their, you know, network effects, their economies of scale in the 1990s and 2000. So this is a form that is contingent on a very particular political, economic, you know, formation. It's not you know, it's not Europe's surfeit of regulation. That's the problem. It's not a lack of innovation. What we're talking about is a business model. And then affordances built on top of that via data and and infrastructure monopolies. So when we talk about ethical issues, you know, we can we can point to that. But that's not really an ethical issue. That's fundamental to what we're talking about and to the what I would call kind of narrative hijacking. That has meant that every time we're talking about AI, we're basically talking about this very particular set of, you know, this very particular approach that has sort of swallowed up the entire the entire discourse and that is being treated as a natural progression of scientific and human progress, not a contingency of a business model we should be very skeptical of, given the centralized power over our infrastructure, given the centralized power over our ideological, historical and information landscape. And now given the power over our institutional and organizational decision making, that is being slowly, then quickly seeded as we integrate these models more deeply into our corporations, our governments and our institutions. So maybe that would be my answer.
Thank you.
Can I pick up on something?
Yeah, please go ahead.
I think it's really important in these conversations. There's such a conflation between risk and trust and talking about co-opting the narrative. I wish we could actually take the word AI and trust out of the equation. You know, it's funny, when you walk down the promenade the number of signs AI and trust they're not going to write AI and risk. But trust in itself is the way I define it. It's a confident relationship with the unknown, right? Like we shouldn't actually have trust in these systems. Like I want to know how these systems work. I want to know how they make money. And so it's like we jumped to the trust conversation because that's very convenient for the platforms before we really had the risk conversation, the mitigation, the management of that risk. Before we even talk about trust.
Let's get into the risk conversation, Matt. Maybe.
Yeah, just following up on both of those. The risk and trust question, I think it's important to, not overcomplicate this and remember how good we have been, how how how we have again and again and again tackled risk and trust issues in every other industry in the past. You know, we want to trust in our medicines that we buy. We don't we want to reduce the risks. How do we do that? Well, first we used to have a completely unregulated free for all in America and elsewhere. And then there were things like the thalidomide drug that was sold to pregnant mothers, promising it was going to reduce morning nausea. And it caused over 100,000 American babies to be born without arms or legs. And that created so much outrage, together with similar scandals that we created the FDA. So now there are safety standards before you can sell a medicine. It's the company's job to deal with the risk trust stuff. They have to make the safety case. And then some government appointed domain experts, which have no money in the game, get to decide is this something they can sell? We do that with restaurants so we don't have to worry about salmonella. Every time we go out for something nice to eat, we do it with cars, with crash testing, we do it with airplanes. Do it with every other industry except AI. It's really not complicated. We've done it so many times before. And, as soon as we have the safety standards in place in an industry, then this amazing innovation that the invisible hand of capitalism creates makes people in the companies put their best talents in to make things safer. You know, if you go to if you compare a biotech company today in Switzerland has some amazing biotech with the big AI company, there's a big structural difference. And the biotech company, they have a much larger fraction of their of their spending going into product safety. How do you reduce the side effects? How do you make et cetera. ET cetera. Then in a big AI company where it's maybe 1% of the people working on safety, sort of as an afterthought and I think, It's very simple. We just need these binding safety standards. We know how to do it, and then companies will innovate to make more trustworthy products so that they can sell them.
Yeah.
I mean, it's like the way I think about the distinction is like sort of risk. Whatever industry you pick, it's like probability, right. So you have you have a commitment to reduce the likelihood of causing harm and bad things happening. And once you get that probability right, you can then go to the possibility. Yeah. And but I truly don't understand how we jumped to the place of possibility without the reduction of harm. And how, how how have are they getting away with it? Why aren't those guardrails in place? Because we shouldn't be talking about what is possible until we figure out that probability piece.
I think it's just a knee jerk reaction of any industry to try to resist regulation initially. And then you get all this hyperbole, oh, it's going to destroy the industry as biotech been destroyed? Of course not. It's thriving. Right. And have restaurants been destroyed because you have to have a health inspector check your kitchen. Of course not. And so it's natural that companies are also trying to resist regulation now, just like they even there was a massive lobbying campaign against seatbelts in the US, you know. And then when the seatbelt law came, car sales exploded because people felt more trust in cars. So I think it's not surprising at all that AI is the Wild West right now, just because it's the new kid on the block, and the quicker we fix it and treat it like other industries, the sooner we'll get the upsides without without the downsides.
There have been a number of people who've just joined us in the room, and I just want to point out this is a town hall. It's designed to be interactive. There's a Slido poll that you can access via these QR codes if you'd like. And we're going to get to audience questions here in just a minute. But I want to ask something for for each of you. There were a lot of discussions around AI safety, a couple of years ago, and that seems like it's really fallen out of fashion. Even even within the big companies themselves. Like like they were talking about it and now they no longer are. I don't know if it was how you you've had a view inside some of these companies yourself. I don't know how genuine those discussions were, but even things like the UK Safety Institute being renamed the what is it, the the UK AI Security Institute now, why did that discussion change so much?
Well, I think actually addressing a lot of these issues would be really, you know, it would it would undermine a lot of business models. And if you look at CapEx in AI, you get a really kind of concrete material answer. There's a huge amount of investment going in betting on a future where these technologies have accelerated into every corner of our lives, every infrastructure, every organization. And you have, you know, the US GDP wrapped up in this. You know, you have, you know, look at the the level of investment and then you see CapEx up there. But revenue is not doing much to meet CapEx. And I think there is a lot of pressure. If you look at the lobbying that is happening from these companies, you know, everyone in Brussels, everyone in K Street is kind of working, you know, four or retained by these companies. And then you have a geopolitical situation, which it is very clear that there are AI haves and AI have nots, and guarding those infrastructures as a tool of geopolitical power has become a, you know, sort of tactic that is being deployed by the governments who have these tools, which also includes bolstering a narrative of inevitability, of acceleration, and of sort of a race to the to AI dominance that is used, I think, conveniently by these companies and some others in good faith, to push back on any sort of regulatory intervention that would, you know, be perceived to slow this down. Now, we do have to ask, like, what are we racing toward? A race to the bottom is not a race we want to win. And I think a lot of these constructs and a lot of these narrative frameworks again, need to be, you know, need to be pushed on. Right. What is AI? AI right now is deep learning. That is a derivative of a very particular platform business model that pulls data in the hands. ET cetera. ET cetera. What does it do? Well, okay, well, Llms do a few things well, but they're actually ill suited for many, many other things. There might be other techniques that are more interesting if you actually measure the success of these technologies when integrated into specific verticals gaming, automotive, other industrial sectors like we need to get much more empirical and the kind of people who would tear a PNL statement apart in a board meeting who would, like, look at that and be like, where did that number come from? Why are we using bespoke accounting methods on that? You know, like really incisive, seem to just let down their entire guards like get almost like hypnotized when they're presented with an AI strategy because we're just kind of assuming we're going to be behind and we're not going to have access to the magical genie that will transform our industry into productivity land. But none of that is empirically based. If you actually look at the data, if you actually start measuring what matters across these ecosystems.
Max. Yeah.
Yeah, completely excellent points. I think in addition, you know, the fact that there is such massive amount of money. In, In the spending, of course, means not only that, there's a bunch of lobbying trying to downplay safety and discredit safety or whatever, but it also has, this more obvious, straightforward effects, like if I notice a number of my colleagues at MIT have stopped talking much about risks now because they have funding from big tech companies. And like, I want to make sure my funding is renewed. We've seen this with the tobacco industry and so many other industries in the past. That doesn't mean that there are that the safety issues have gone away in any way. In fact, contrariwise, so many of the things that people warned about five, ten years ago are actually happening now. We see AI that lies we we have. There was this great work by anthropic, for example, showing how how AI when it was told that it was going to be shut down, it found in the email server that the the guy who was in charge of shutting it down was having an affair and had blackmailed him to not do it. This was in simulation, but, you know, the the concerns are just as strong as ever. It's been known for a long time also that AI would one day become really good at manipulating people. Last month, I spoke with Megan Garcia, whose son committed suicide after being groomed by a chatbot for six months. And it's just as a parent, you know, it's just so heart wrenching to see that exactly the things we could have prevented if we had taken these safety issues more seriously are actually happening now. But I think on the on the upside, the good news is particularly the child harm that we're beginning to see, you know, everywhere from anything from non-consensual child abuse, deepfakes to suicides and psychosis is really, really resonating. I've never seen any issue before that made the broader public so engaged in AI risk. And we now have this last few months, this crazy broad coalition in the United States, I call it the Bernie to Bannon coalition, the B2B coalition.
Tell us about superintelligence.
Yeah, yeah, we had that's just part of it. We we did a statement saying that, we shouldn't build superintelligence until there's a scientific consensus, broad scientific consensus that we can control it and so on. And the people who signed it, Steve Bannon signed it. A bunch of hardcore Democrats signed it, bunch of top military people signed it. You know, Mike Mullen was the number one, the head of the Joint Chiefs of Staff under two presidents. They signed it. What do all these people have in common? Even many people thought they wouldn't agree on anything. A lot of faith leaders, and they all agree that, you know, we want a pro-human future where the AI creates a good future for people, not a bunch of child suicides and other crazy stuff. We're trying to make a great future for the humans, not for the machines. So of course all these people are going to agree. If we got invaded by some weirdo aliens that don't care about humans, then of course Republicans and Democrats and Americans and Chinese would also join together. So there is this is, to me, very encouraging, actually, that there was a poll showing 95% of Americans now don't want an unregulated race, just craziness. And I think there's real potential that we translate that into what I said in the beginning, just treating AI, the AI industry, like every other industry, saying, you know, there are safety standards for everyone else here are yours as well. And, and and then, we'll have the incentives so that people, even the people who just look at the balance sheets, will prioritize making things trustworthy, secure and safe.
I want to that seems like a good point to see. Can we see what the consensus in the room is around the pole on the Slido poll? Do we have those results?
Not bad question.
Yeah. Let's see how people answer. Yeah.
And if you answered trust, let me know why.
39% says I rather don't trust 32. I'd rather trust.
I'd rather.
Trust rather what is that modifier doing in there? Yeah.
It's what's what's clear is like the yeah, the.
Minority view here is I definitely trust, at 8%. I definitely don't trust is also only at 8%. Okay. That's interesting.
What's the breakdown? What's 8%?
Yeah. Can we can we get a show of hands of people who definitely trust. No, nobody wants.
To show.
Don't want to.
Put anyone.
On the spot.
I wouldn't either.
I want to come back to something else that we're the consensus around, because I think it speaks a different AI issue, which is increasingly you're seeing like people on, you know, Republicans, Democrats, red states, blue states in the United States. That is, don't want data centers near them, like the whole thing that, that, that everything that AI runs off of, it's, it's it's from a data center and they're, they're going up all over the country. They're, they're driving especially in places like Virginia and Georgia. They're driving up people's power bills. They're using water in the West. I'm curious if but I think it speaks to a larger the environmental impact of AI. I'm curious if you guys have thoughts on that, on the on the sort of the, the, the footprint that we may be leaving on, on society for years to come with new power plants and things like that.
It's interesting. There's no panel on climate change in AI, not one panel linking those two issues. I find it astonishing. Like, I mean, anyone who was at Davos three years ago, climate change, sustainability, that was I don't even want to call it the shiny toy, but. Right. It was the central theme. There is not one panel like how how I.
Realize that.
Yeah, like whatever we call it sustainability, energy efficiency, climate change. We have to link these two issues. I'm not an expert on data centers, but I do find that amazing that you can have a whole program three years ago around the climate and environment and sustainability and green energy, and then three years later, there is nothing linking those two issues. So I ask, who's driving that agenda? Because that cannot be. Well, I don't know if it's deliberate or if it's unintentional, but to me, that's the glaring problem, is that in society, this new thing comes along. And then another issue, it sits over here and there's not enough people who just join the dots. I can't comment on data centers. It's really not my area of expertise, but I found that really astonishing about human beings and just these discussion forums that we don't do enough linking of those issues. You can't separate those two things. Yeah. Sorry. That was me on my. Yeah.
No, I'm.
Glad you brought it up. But I do think it's. Or maybe I should ask you, do you think it's true that AI is fundamentally extractive technology? I mean, it's pulling from human knowledge. It's it's using our energy systems. Like. Is it is it do you think it's worth it?
I mean, I.
Fundamentally this version of AI, this deep learning monoculture, I'm going to be a pedant about this continually. Yeah. That's that's how it works, right? You pull a bunch of data and that data is sort of stuff that was put on the web or scanned, you know, starting in the sort of early 90s through now. And that's kind of, you know, that's all of human knowledge, folks. Reddit comments. And it's, you know, trained on that distribution of that normal distribution maps to the context of use. Yet it will be more useful. If it doesn't, then you can sort of bolt a Rag database on the side and kind of, you know, try to try to adjust it, but, you know, ultimately that's what it is. It extracts from that. And I think, you know, I come from the arts. I went to art school in high school, I was very serious about it. And seeing the extraction of the creative industries is, is pretty heartbreaking to me. And just the way that creative professionals are being treated and sort of understood in that industry. That's a that's not a comment on copyright. That's just a comment on sort of the values that seem to be encoded in how this is being created. And then, of course, you know, energy wise, the sort of AI monoculture that is scale at all costs on very, very resource intensive deep learning approaches, takes a huge amount of energy, takes a huge amount of water. Data centers, you know, are not always clean. You have a lot of issues with dry towns now in the US, you know, energy bills going up. And then, of course, you know, you have green sources of energy coming online, but they're not at all compensating for the increased use. And so a lot of claims of sustainability end up being very paper thin that says, hey, we're using solar for this data center. But that means a number of coal plants haven't been decommissioned that are now continually serving communities. And I think that's, you know, that's an issue that, you know, I kind of feel like some, you know, when it's in the far future, we can all talk about it or the farther future, it's like, yeah, let's have a panel on that. But, you know, this is rubber meets the road, right? You have people, you know, have you have brownouts that are happening now. You have thresholds that are being reached. And so the action that would need to be taken is, you know, significant. And I think we sort of, you know, the sad truth is when we frame these things as ethics or nice to have or sort of like, you know, decoration on the side of an inevitability, then we draw back from actually addressing those issues, because addressing them would require looking closely at that narrative and realizing nothing. None of this is inevitable. All of this could be changed. And if we prioritize sustainability and, you know, making sure that we're using our finite resources in a way that is socially beneficial, we might make different choices.
Max is something you have thoughts on?
Yeah, my opinion is my take is that the water issue is actually smaller than some people have claimed, but the energy issue is very real. And on the on the data issue. Oh my gosh. I mean, you're really making my point there, Meredith, about how again, AI industry is treated completely different from any other industry. What would happen if if some, some company in the film industry just blatantly ripped off everybody's copyright, you know, yet for some reason, companies have been allowed to just take every single copyrighted book and whatever. You know, I find my books even in training data of and just do whatever they want with it. And but just because, oh, it's AI exceptions. We were special, you know, and yeah, there was a 1.5 billion copyright settlement against anthropic now. But it's sort of that's just the tip of the iceberg, really. Why I think there should be a law saying that companies have to declare what they put into their training. If you buy a food product in the supermarket, you want to know what's in it, you know. And as soon as you did that, companies would get their pants sued off from all the copyright infringements. Now they it's hard to sue them because they won't tell you what they, what they what they trained on. And again, I don't see any reason why we need to why we should have this kind of discrimination where all the other companies have to play by rules and AI companies don't.
So I want to get to some audience questions here in a moment. But before I do, let me leave or get you guys to explore one thing, which is we're talking about some negatives here, but I'd love to know, like what are your thoughts on what can we do as individuals to increase our agency to, to to ensure that there are rules or regulations to ensure that, that we can trust these systems? What what is there to do?
Well, I'll talk from a personal level. I think it's really important to set boundaries in your own work. Right. So, to your point, I mean, I could use any one of these tools and it could write my next book because I've trained it so well. And sometimes it's like I'm writing an article and I think, well, what would it say? And I think, oh, that's really good. Maybe I should just use that, right? Like it's so easy to slip into that space. So my rule and this is a really simple one, is that when I'm drafting something I'm away from my computer. I use a pen and paper. I have to complete a first draft with my own thinking and my own points. It's a very simple boundary and rule. It is so easy to break when you're under pressure. And then I come back and then I ask it questions, and I ask it to go back into my whole history of work. And it does pull out really interesting points and it does improve the argument. But if I didn't set that own that boundary, and I started at the point of, I've got to write an article for you, Matt, for the MIT, I've only got an hour. Could you draft something on trust and ethics and AI? And then I could tinker with it and you probably would not know. Right. And then I could say, could you edit it in Matt's voice and you'd be like, this is really clean copy. Thanks so much, Rachel. Now, so this is this is the thing, right? For me, the boundaries are actually trust in ourself as well. Like, that's the thing. You can control you. That's the agency piece. And the piece that we often don't talk about is identity. So I just want to give you one brief example, because it's a study I came across and it's from the 70s, and there was a big sociology done on bakers. Now bear with me. And it was when machines were replacing bakers in bakeries. So, you know, they wouldn't need the bread anymore. They'd press the croissant button and out would pop croissants. And at first the bakers loved it. Right. They'd go home earlier, there were less burns. They were far more efficient. And then what the sociologists found was in a very short space of time, within six months, they said they couldn't call themselves bakers anymore. It was the identity piece that went, and they said they would go home to bake bread. And I give that as an analogy of how quickly that can happen. If you don't think about who am I, right? Am I an artist? Am I a writer? Am I a teacher? Like, what is that real meaning I have in my life? And how does that relate to my thinking and create some kind of personal boundaries around that? Because that outsourcing is so good and it's so quick and it's so easy that unless we set those guardrails for ourselves, how do we pass them on to our students? How do we pass them on to children? We have to understand how to carry that line, to pass it on to others.
Well, I know there's a question right back here because you had your hand up. Can we get a microphone over. Over here?
Thanks. No, I was really moved by what Rachel was saying about the probability and possibility. And yesterday I was at this panel with, Joel Kaplan from meta and then minister of deregulation from Argentina, who are probably best buddies by now. And they were talking about like, the only thing we need right now is societies is just to let AI run things. And then if harm happens, we, like, figure it out, but let it run for a while, just like. And then we will kind of like see what harms are there. But then like nobody mentioned the threshold for harm, so nobody mentioned like, well, if we have these signals, this is what we are monitoring. Nobody mentions what people are actually looking into. There's just this like push for like just keep going. The same message came from from Jason Huang from Nvidia saying, we just need more investment in infrastructure and everything else is going to figure itself out. So how do we kind of create these pressures for the things that we want to see in terms of safety? Because you're mentioning the coalitions, you know, we're talking about? Well, we know the business models are broken and they're not really figuring out the humanity side of this. So what what will create because we are sitting in these rooms separately from those people and talking about ethics as decoration on the side, but like, how will we do this? Like what is what is the change?
Either of you go ahead. Yeah, yeah.
Let's get super concrete. Like if we if we were to treat AI like any other industry, what would actually happen and how would how would this affect what the companies do. So let's suppose like you know, Microsoft has a new some new productivity tool. There's some safety standards. They will look at that probably the same way the Food and Drug Administration would look at a new kind of fruit juice, saying, you know, this product, random Microsoft productivity tool, very hard to see much potential for harm for the fruit juice. You just want to make sure it doesn't have arsenic in it or whatever, but that can be handled just with some random. It would be basically perform a super quick. It's out in the market right now comes a character AI. Here is our new AI girlfriend for or some other company you know, for 12 year olds that would be looked at more like maybe a new opioid drug. High, high potential for addiction. Pretty obvious how it might cause harm, how it might cause deaths. Okay, off to the clinical trial. You go. They do a control group, a test group. And notice that actually there's a strong increase in suicidal ideation and the those who tried this chatbot. And so you can see it from the chat logs pretty quickly. So sorry. No approval to you next customer please. And these sort of or now someone comes along and says, hey, I want to do this recursive self-improvement with no human oversight. So, oh, that sounds a lot like digital gain of function research. So how do we do this in biology? Oh, in biological gain of function research actually isn't allowed in the US right now. So next customer please. It would very quickly be noticed by the companies that these are the kind of products where approval is trivial. And you can make a lot of money, resources get shifted into their, and frankly, AI girlfriends is is not. I'm quite sure that's never been Microsoft's key idea for how you're going to make money in the future anyway, right? Shrink into this little fringe thing that it should be so we don't have to lecture the companies at all. We just have to have these rules that treat them like other companies. And then I think we're going to see a massive behavior change in the companies. Well, the government, of course, just like the FDA, you know, the government decided we don't want another thalidomide.
Max, I come back to what Meredith said. I mean, I'm all for the rules and the regulations, but this is a market problem. Like the world. The economies will collapse. I love that he needs more investment in infrastructure. That's wonderful. Right? But like.
Biotech did not collapse.
Biotech is struggling now. However, it's really hard.
But it. didn't collapse in the 60s when the was created.
Well, right.
And I think I think it's increased trust in medicines and massively increased.
Biotech company. I can you know I'm not going to name any names, but they're doing incredible AI centric research on cancer. It's high risk, high reward. They're using DeepMind's protein folding database extremely hard. This is long term research. It may not pan out, but it's you know, that is a social good QED. They can't get series B because the VCs want a chatbot to cash out. Right. That's what we're looking at here. I mean, you know, that's a that's one story among many that you hear in this ecosystem. And it's, you know, again, this is a monoculture. I think like I do want to answer this question because I just find it I find the you know, I think I worked on my first sort of, you know, AI regulation, you know, like, how would we think about these guardrails around machine learning and probably 2015. Right. And that was right after the DeepMind acquisition at Google 2012 was AlexNet. And that's where everyone started getting interested in deep learning again. So this is pretty early. I've probably had this conversation once every quarter since then. What are we going to do? Like, you know, and I don't there's in some sense it's gotten worse and worse. The level of discourse has gotten more and more sort of like baby brained, to be blunt about it. Like, yeah, we just accelerated all wash out in, in in the wash. And meanwhile I'm I'm leading signal. How many people here use signal? Okay. I love you all. You know, but signal is core infrastructure to preserve the human right for private communication. It is used by militaries. It is used by governments. It is used by boardrooms. It is used by dissidents. Ukraine uses it extensively. This is these are contexts where the ability to keep your communications private is life or death. Signal is an application built on on top of operating systems. We build one version for Android, one version for iOS, one version for windows, and you know, Mac OS on your desktop. And we take this responsibility really seriously. We open source all of our code so people can look at it. We open source our cryptographic protocol and our implementation. We're working with some people with with Max and some folks to formally verify that, which means mathematically, you can prove that what it says in the code is what it's doing. And that's a level of assurance we put out there, because we recognize that this could, you know, people die if we mess this up, but we have to run on top of these, these, these operating systems. And you are seeing the three operating system vendors now rushing to integrate what they're calling AI agents into these operating systems. Now, AI agents, the marketing promise is you have a magic genie that's going to do your living for you, right? So you ask the AI agent, hey, can you plan a birthday party for me and coordinate with my friends? That sounds great. You can put your brain in a jar and you don't have to do any living. You can just walk down the promenade, you know, seagulls in your mind. And while the agent plans your birthday party. But what is it an agent actually doing at the level of technology? In that case, it would need access to your calendar, access to your credit card, access to your browser to simulate mouse clicks and place orders. And in this hypothetical scenario, access to your signal messages to text your friend as if it were you and to coordinate and then sort of put that on your calendar. Right. That is a significant security vulnerability. So while we're telling stories of inevitable magic genies that exist in a bottle, what these agents are doing is getting what's akin to root permission in your operating system. For the Unix people in the room, they are reading data from your screen buffer at a pixel level, bypassing signal. They are hooking into your accessibility APIs to get audio to get image data from the screen. They are rooting through your file system, the sort of deepest level of your operating system that controls what the computer can and cannot do, what software can and cannot do, what data you can access, and what you can do with it. They are making remote API calls to third party services. They are sending all of that to the cloud, because there is no LLM small enough to run on your device, and they are processing in the cloud to create statistical models of your behavior so they can guess what you might want to do next. These are incredibly susceptible to prompt injection attacks. There is no solution for that because when it comes to natural language, which is what these systems are processing, they fundamentally cannot tell the difference between an authentic desire of the user and language. That may look like an authentic desire, but is in fact not representative of what people actually want. This if you look at the architectures of these agents on the operating system, they look a little bit like the architectures of targeted malware given the permissions that they allow, given the data access they allow and given the vectors to, you know, send that off off your device process in the cloud, you know, access a website outside, etc. this is an existential threat to signal to the ability to build securely and privately, privately at the application layer. And it is a fundamental paradigm shift in the history of computing, where for decades and decades we viewed the operating system as a neutral set of tools that developers and users can use to control the fundamentals of how their device worked and how the software worked. These are now being effectively sort of remote, controlled by the companies building these agents and these systems that are ultimately taking agency away from both developers like signal and the users of these systems. Again, I went into this level of technical detail because, you know, that's the level at which any dignified adult conversation should be happening. Like, come on, you think you can take responsibility for the entire world, but you don't have to answer for it? I did not come up in a world where that had any dignity to it. So I think, you know, like in some sense we just need to demand a bit more spitefulness of the people who claim to own the future and recognize this is what's happening. If you're hooking into my accessibility API, you have ultimately decimated signals ability to provide this existential, this this fundamental service. And I will shut it down before I will continue to operate without integrity, because I do not want people being harmed because they trust us. They trust us to provide a service we no longer can.
Well, that was wow. Okay. Thank you. And with that, we're out of time. I'm sorry. I know we had some more questions in the room we didn't get to, but thank you so much for joining us today. Thank you for being here. I hope you all got a lot out of that I did. Thanks, everybody.
Thank you, thank you.
Thanks.